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ABSTRACT 

Wo  consider  the  evaluation  of  and  hounds  for  the  rate  distortion  functions 
of  independent  and  identically  distributed  (i.i.d.)  sources  under  a magnitude- 
error  criterion.  By  refining  the  ingeneous  approach  of  Tun  and  Yao  we  evalu- 
ate explicitly  the  rate  distortion  functions  of  larger  classes  of  i.i.d. 
sources  and  we  obtain  famiLieis  of  lower  bounds  for  arbitrary  i.i.d.  sources 


I.  I.YIROuUTlON 


The  rate  distortion  function  of  an  independent  and  identically  distributed 
(i.i.d.)  source  is  clearly  equal  to  the  rate  distortion  function  of  each  random 
variable  of  the  source.  We  will  evaluate  the  rate  distortion  function  of  a 
random  variable  X with  probability  density  p(x)  satisfying  certain  conditions. 

A magnitude-error  criterion  will  be  used  throughout  without  further  reminde'-. 

The  procedure  used  was  introduced  by  Tan  and  Yao  (1975)  and  is  based  on  the 
well-known  analytical  expression  of  R(D)  which  is  stated  for  reference.  (See 
for  instance  Berger,  1971). 


THEOREM  A.  I.et  X be  a random  variable  with  probability  density  function 
p(x)  and  rate  distortion  function  R(D) . For  each  s<0,  let  A he  the  set  of  all 
non-negative  functions  X satisfying 

CO 

Cs(y)  = / As(x)p(x)exp(s|x-y|)dx  < 1 I 

-CO 

for  all  y.  Then  for  all  D > 0, 

CO 

R(D)  = sup  [sD  + / p(x) (x)dxj  . I 

s^0,X  eA  -°° 

s s 

For  each  s<(),a  necessary  and  sufficient  condition  for  X to  realize  the 
supremum  in  (2)  is  the  existence  of  a probability  distribution  G^,  which  is 


related  to  X by 
s 


[\.0)]-1  = / exp(s|x-y|)dG  (y) 


is  such  that  Cs(y)  = 1 a.e.  [dG  ] . Moreoeve r,  for  such  X^  and  Gg , R(l') 


is  given  parametrically  in  s by 


R(D^)  - sl^  + / p(x)£nXs(x)dx 

-id  ‘ s<l 

O'  on 

I>s  = / / X^(x)p(x)|  x-y|  cxp(s|  x-y(  )dx  dG^ty) 


ingeneous  procedure  to  search  for  X satisfying  (1)  and  (5)  was  given 


by  Tan  and  Yao  (1975)  and  is  described  in  the  following  immediate  corollary 
of  Theorem  A. 

COROIjLARY  A.l  [Tart  and  Yao  (1975)].  Let  X be  a random  variable  with  probability 
density  function  p(x)  which  vanishes  outside  the  interval  (a,b),  -co<a<b<co.  For 
each  s<0,  let  V be  a subinterval  of  (a,b)  and  assume  that  the  distribution  func- 
tion Gs(y),  y£Vs>  whose  total  probability  is  concentrated  on  Vg , and  As(x), 
xe [a,b] , satisfy 

[A  (x)]"1  = / exp(s|x-y|) dG  (y) , for  all  xe[a,b]  (6) 

5 V 

and  S 

b 

/ Ag (x)p (x) exp (s | x-y | ) dx  = 1 , for  all  yeVg  . (7) 

a 

If  A^  satisfies  (1),  then  the  rate  distortion  function  of  X is  given  by  (4)  and  (5). 

The  significance  of  this  straightforward  corollary  lies  in  the  fact  that  for 
some  densities  p,  intelligent  (or  appropriate)  choices  of  Vs  can  be  made  such 
that  (6)  and  (7)  can  be  solved  and  the  solutions  satisfy  the  properties  stated  in 
the  Corollary.  Using  this  procedure,  the  rate  distortion  functions  of  an  i.i.d. 
Gaussian  source,  and  of  a certain  class  of  i.i.d.  sources  were  calculated  ex- 
plicitly [Tan  and  Yao,  (1975)]. 

In  this  paper  we  make  two  uses  of  this  procedure  of  Tan  and  Yao.  First,  in 
section  II,  we  refine  their  results,  by  a substantial  weakening  of  the  conditions  on 
the  density,  thus  calculating  explicitly  the  rate  distortion  functions  of  larger 
classes  of  i.i.d.  sources.  In  Theorem  1 the  density  of  the  source  has  finite 
support,  Theorem  2 treats  concave  source  densities,  and  in  Theorem  1'  the  support 
of  the  source  density  is  the  entire  real  line  or  a half  line.  Secondly,  we 


develop  a family  of  lower  bounds  for  the  rate  distortion  function  of  an  arbitrary 
i.i.d.  source  (Theorem  3)  and  compare  them  with  the  Shannon  lower  bound  in 


Section  III.  We  also  indicate  how  Theorems  I and  1'  may  be  combined  with  a 


known  approximate  result  (Theorem  B)  in  evaluating  the  rate  distortion  functions 
of  certain  i.i.d.  sources  whose  densities  do  not  satisfy  the  assumptions  of 
Theorems  1 and  1'. 


II.  RATE  DISTORTION  FUNCTIONS  OF  I.I.D.  SOURCES 

We  first  consider  rate  distortion  functions  of  random  variables  with  con- 
tinuous densities  which  vanish  outside  a finite  interval. 


THEOREM  1.  Let  X be  a random  variable  with  probability  density  function 
p(x)  which  vanishes  outside  the  interval  [a,b]  , < a < b < 00 . Assume  the 

following: 

(A)  p is  continuous  with  median  p and  there  is  an  at  most  finite  set  of 
points  a = d/1<d,<...<d  <d  , = b (mao')  such  that  on  each  [d.,d. 
j=0, 1 , . . . ,m,  p(xl  is  differentiable  and  its  derivative  p' (x)  is  absolutely 
continuous  and  satisfies  p'(dj)  > p^(d^),  j=l,...,m,  where  p'(dj)  and  p|(dj) 
are  the  left  and  right  limits  of  p'  at  dj  respectively . Also 

x b 

/ p(t)dt  > 0 for  x > a;  / p(t)dt  > 0 for  x < b . 
a x 


(B)  The  function 


b 


K2(x)  = p(x)/  / p (t) dt  for  xe[u,  b) 
x 

diverges  to  +0°  as  x increases  to  b;  and  the  function 

x 


K2(x)  = p(x)/  / p (t) dt  for  xe (a ,p] 
a 


(8) 


(9) 


diverges  to  +°°  as  x decreases  to  a. 

Then  for  each  se  (-°°,-2p(p) ) , there  exist  unique  a > 0 and 

that  a ip-a  and  b tb-p  as  s^-03  and  a and  b arc  determined  by 
s s s s ■ 


> 


0 such 


p-a  . = min  {ye  (a, pi : K,(y)  = |s|} 
U+l>s  = max{ye(p,b1:  K j ( y ) = |s[}  . 


Suppose  in  addition  that 


(C)  for  each  se  (-00,  - 2p  (p)  ) , 


p(x)  - s >"(x)  5 0 a.e.  [Leb. ] on  [p-a^.p+b^] 


Then  the  rate  distortion  function  R(D)  , 0 < D < D , of  X is  qiven  paranyetrically 

max 

in  s by 


R(0s) 


I • I - 

= £n ■ ^ - / S p(x)£n(ep(x) )dx  - 


£n(p(p-as))/  Sp  (x)  dx 


- £n(p(p+bj)  / p ( x)  dx 


" s p-a  b 

D = -nrr  f p(x)dx  + / s (p-as-x)p(x)dx  + J (x-p-bs)p(x)dx 
I ‘ ' p-a  a ' neb 


where  -m  < s < -2p(p)  and  D = / |x-p|p(x)dx  . 

max  a 

Proof : We  will  show  that  all  conditions  of  Corollary  A.l  are  satisfied  with 

V = [p-a  .p+b^.]  where  the  dependence  of  ag,  b^  on  s will  be  specified  later. 

Substituting  (6)  into  (7)  and  writing  the  integral  from  a to  b as  the  sum 
of  the  integrals  from  a to  p-a^.  to  p+b^  to  b,  we  have 

A exp(sy)  + A.,  exp(-sy)  + / a (x)exp(s | x-y | )dx  = 1 ( 

I , S i.  ,s  ,,  s 


for  ycV  , where  A.  , A_  and  a are  given  by 
s’  1 ,s  2, s s * J 

P-a 

A,  = / sp(t)dt/  / exp(st)dC  (t) 

a V 

s 


Ufa) 


b 


A_  . = / p C t ) dt / J exp (-s  t ) dG  ( t ) 

,s  p+b  V 

S S 

&s ( x ) - p(x)/  / exp(s|x-t|)dO  (t)  . 

V 

s 

By  differentiating  (14)  with  respect  to  y,  we  find  that  it  is  necessary  that 
cts(x)  = |s|/2,  Aj  s = ^exp(-s(u-as))  , A,  s = ^exp(s  (p*bs) ) 


(17! 


and  these  are  also  sufficient  for  (14)  (which  can  be  verified  by  substituting 
into  (14)) . 

Substituting  the  solutions  into  (15),  (16)  and  (17)  we  obtain 


U-a. 


/ exp(s (t-p+a  ) )dG  (t)  =2  / hp(t)dt 


(IS) 


p(s(p+b  -t))dG  (t)  = 2 / p(t)dt 

y+bs 


(19) 


/ exp(sjx-t) )dC  (t)  = (2/[s|)p(x)  xeV  . 


(20) 


Clearly  (L8)  and  (19)  are  consistent  with  (20)  if  and  only  if 


and 


p-a 

/ Sp (t) dt  = -j-y  p (p-a  ) 
a I b I 


Jb  p(t)dt  = p(p+b  ) . 

p+b  I i 


(21) 


(22) 


Now  (21)  is  equivalent  to  k'  (p-a  ) = |s|.  Note  that  conditions  (A)  and  (B)  inply 

that  k.,(x)  is  continuous  on  (a,p] , differentiable  on  (a,p|  except  at  those  d.’s 

which  belong  to  (a,pj  at  which  left  and  right  derivatives  exist,  and  satisfies 

K,(p)  - 2p(p)  and  lira  k0(x)  = +°°.  It  follows  that  given  any  se -2p(p) ) the 

xla 

equation  k-,(p-a  ) = |s|  has  at  least  one  solution.  For  reasons  which  will 


become  clear  later  on  in  this  proof  we  will  choose  the  smallest  solution: 


* 


li  - a min{y<  (a,y):  K .,(>•)  |sj( 


which  is  clearly  such  that  y-a  +a  as  si-00  and  has  the  following  properties  (.to 
be  used  later  on): 

K +(y-a^)  < 0,  and  K,(y)  > |s|  for  all  \v  (a,y-a  ) . 

Similarly,  by  the  properties  of  kj(x),  is  uniquely  determined  by 
U + b^  = maxfye (y,b) : Kj  (y)  = |s|} 


(24) 


and  has  the  following  properties:  y+b.tb  as  si-0"  , 

kj  (y+b  ) -•  0,  and  K^(y)  > |s|  for  all  yt (y+bs ,b)  . 

We  next  show  that  for  each  sc (-tu,-2p(y) ) , the  distribution  function  Gs(x) 

_ 9 

which  has  absolutely  continuous  part  with  density  p(x)-s  ~p"(x)  on  V and  zero 
elsewhere,  discrete  part  with  atoms  at  the  points  p-a^ , p+b^.  and  the  d.’s  which 
are  in  [y-a^.,  y+b^l  and  masses  to  be  determined,  and  zero  continuous  singular 
part,  is  a solution  of  (20).  For  notational  convenience  we  will  work  with  the 
"density"  g^  of  the  above  described  distribution  function  Gs  which  is  thus  of  the 
form 


_ ? 


£+n-  1 


IL(t)  = p(t)-s  p"(t)+C  6 (t-p+a  )+C9  6(t-p-b  ) + J D S(t-d.) 

1 ,b  b j _ £ J > ^ 

for  t c V and  zero  elsewhere,  where 
s 

d„  , < y-a  < d.<...<  d.  . < y+b  < d, 

£-1  s i £+n-l  s £+n 

and  fi(*)  is  the  Dirac  Delta  function. 

The  masses  C,  , CL  and  D.  can  now  be  determined  so  that  (20)  will  be 
1 ,s  2,s  j ,s 

satisfied.  We  find  (see  Appendix  1) 

ci,s  = Is!  ~Nslp(y-as')  - p{(y-‘VI  » 

(:2,s  = Is!  ’ 1 1 s |p(y+i,sl  + p'(y-i>s)l  » 

M Ip'  (dj ) - p[(d.)  | , c j +n- 1 . 


(25) 


(2(0 


D. 

.1  »s 


Having  determined  g.  so  as  to  satisfy  (20)  it  remains  to  he  shown  that  g is  a 
probability  "density"  function,  i.e.  that  C is  a probability  distribution  func 

_ 9 

tion.  Since  p ( t > - s ~p"(t)  > 0 a.e.  [LebJ  on  V.  by  assumption  (C)  and 
p ' (d . ) - p|(djt  s 0 by  assumption  (A),  it  is  clear  from  t he  expressions  (25)  and 
(26)  that  G is  a distribution  function  if  and  only  if 


|s|p(y-as)  - p|(y-as)  > 0 

(27) 

|s|p(y+bs)  + p’(y+bs)  > 0 

(28) 

/ g (t)dt  = / dG  (t)  = 1 . 

V ' vs 

(29) 

To  show  (27),  we  proceed  as  follows.  Since  ^(p-a^)  < 0 we  have  from  (9) 


d-a  9 

p|(p-as)  / sp(t)dt  - p (y-as)  < 0 
a 

and  using  (21)  obtain 

y-a. 

I / Sp(t)dt|  [p|(p-as)  - |s|p(vi-as)]  < 0 . 
a 

M-a 

Now  since  a < y-a  we  have  / "p(t)dt  > 0 by  condition  (A)  and  thus  (27)  follows. 

a 

(28)  can  be  proved  similarly,  and  (29)  can  be  verified  easily. 

Next  we  need  to  show  that  (y)  < 1 for  y^V^.  A^_(x)  f°un^  by  substituting 

g^  into  (6)  and  we  have  (see  Appendix  2) 


As(x) 


[ | s |/2p(yi-as)  lexp(-s(p-as)+sx) 
|s|/2p(x) 

[ |s  | /2p(u+bs) ]oxp(s (y+bs)-sx) 


xe [a,y-ag] 


x<_  V 


s 


xc[yrbs,h]  . 


(50) 


Now  suppose  that  a < y < y-a^.  Then 

Cs(yl  = / As(x)p(x)exp(s(y-x))dx 
a 


b 

/ A (x)p(x)exp(s(x-y) Vlx  • 
> 


Pi fferentiat i ng  l\.  (y)  with  respect  to  y,  we  have 


who  re 


/ ‘ ' 

C^(y)  ■ | s ( / A (x)p(x)exp(s(y-\))dx  f |s|  / A (x  )p  (x)exp (s  (x-y ) )dx 

i i a h y 

= |s| (Cs(y)  - hsCy)) 


h (y)  - 2exp(s y) / A (x)p(x)exp(-sx)dx  . 


Substituting  (30)  into  (52)  we  have 


hs(y)  = |s|exp(-s(y-a  )+sy)/  p(x)dx/p(y-a  ) a<y<y-a. 

^ s s 


and  finally,  because  of  (21) 

y 


M-a.. 


hs(y)  = exp(sy)/  p(x)dx/exp(s (y-aj ) / Sp(x)dx  a<y<y-a  . 
a a 

We  now  show  that 


f(>')  = exp(sy)/  p(t)dt 
a 


ye(a,p-as] 


is  increasing.  Indeed  we  have 


.y 


f’(y)  = cxp(sy) (p(y)  - |s|J  p(t)dt)  . 

a 


Now  (23)  and  (9)  imply,  as  was  remarked,  that 


K2(y)  - K9(y-as)  = | s | for  aCySy-a^  . 

It  then  follows  from  (9)  that  f 1 (y)  > 0,  acy'Sy-a^.,  and  thus  f is  increasing  on 
(a,y-a  ].  Hence  h (y)  < 1 for  ye(a,y-al  and  since  h (a)  = 0,  it  follows  that 

5 5 5 S 

h.(y)  < 1 for  ye [a, y-a  ]. 

^ s 

Now,  as  in  Appendix  A of  [Tan  and  Yao,  (1975)]  it  is  shown  that  C (y)  < 1 
for  all  y^V  . 

Thus,  by  Corollary  A.l,  R(D)  is  given  parametrically  by  (4)  and  (5).  (4) 


(32) 


and  (5)  arc  shown  in  Appendix  3 to  have  the  final  expressions  given  in  (12)  and  (13). 
This  completes  the  proof  of  the  Theorem.  i j 


] 0 


The  following  examples  are  applications  of  Theorem  !. 

Example  I.  Let  p(x)  be  a truncated  double-exponential  density  function  defined 
on  f-c,c] , c > 0,  i.e. 

p(x)  = aexp(-a|x| )/2(l-exp(-ac))  |x|sc,  a>0  . 

The  assumptions  of  Theorem  1 are  satisfied  and  are  verified  in  the  following: 

(A)  p(x)  is  clearly  continuous  with  median  p = 0.  p(x)  has  continuous 
second  derivative  everywhere  except  at  x - 0 at  which  the  first  derivative  is  dis- 
continuous and 

p^(0)  = a‘'/2(l-exp(-ac) ) > pj(0)  = -a2/2(l-exp(-ac))  . 

(B)  K^(x)  = a exp  (-ax)/ [exp  (-ax)  - exp(-ac)  ] t+°°  as  xtc  . Similarly, 

K7(x)f+°°  as  xi-c.  In  fact,  K^(x)  is  easily  seen  to  be  monotonical ly  increasing 
for  xe[0,c]  and  ^(x)  monotonically  decreasing  for  xe[-c,0].  Thus  (10)  and  (11) 
give 

ag  = c + a *&n(l-a/|s | ) . 

Now  |s|  takes  on  its  minimum  when  = 0.  Tliis  implies  that 
| s | 5-  a/  [ 1 -exp(-ac)  ] > a.  Thus 

(C)  p(x)  - s 2p"(x)  = (l-a2s  2)p(x)  > 0 for  se  (-°°,-a/ [ l-exp(-ac)  ] and  for  all 
xe  ["as,asl  - 

Therefore  R(D)  , 0 < D < D is  given  by  (12)  and  (13).  Calculation  (routine 
and  thus  omitted)  shows  that 

R(D  ) = Hn [ | s | ( 1 -exp(-ac) ) /a]  - a exp(-ac) [c+a  \n(l-a/| s | ) |/ [l-exp(-ac) ] (331 

Ds  = r | s | ~ 1 + a"lexp(-ac)£n(l~a/|s|) |/[ l-exp(-ac) ] (34) 

where  j s [t  [a/(l-exp(-ac))  ,°°)  and 

l>  r:  fa  ^ - (c+a  Scxp(-ac)  ]/ [l-exp(-ac)  ] . (33) 


-A 


1 

Example  2.  Let  X be  a random  variable  with  density 

p(x)  = 1 / (x  2. nl 00)  , 0.01  < x < 1. 

Then  p(x)  is  continuous  and  differentiable  with  p = 0.1.  Conditions  (A)  and  (S) 
are  clearly  satisfied.  Note  that  K^(x)  decreases  for  p < x f e * and  then  increases 
to  +00  as  xfl . Condition  (C)  is  not  satisfied  for  all  s but  only  for  some  s in 
(-°°,  -2p(p)).  In  this  case,  only  a portion  of  R(D)  can  be  obtained  (corresponding 

to  those  for  which  s satisfies  (C) ) . We  have 

p(x)  - p"  (x) / s 2 = p(x) (l-2/s2x2)  i 0 

if  and  only  if  x > /2  / } s | . Thus  only  for  large  |s|  (C)  will  be  satisfied.  for 
s = -72.135,  we  have  /2/|s|  = .0196  and  from  (10)  y - a = 0.02.  Thus  for 
s < -72.135  (C)  is  satisfied.  This  portion  of  s corresponds  to  a region  of  small 
distortion  D (since  s is  the  slope  of  R(D))  and  for  this  region  R(D)  is  given 
parametricaly  by  (12)  and  (13). 

We  now  show  that  the  class  of  continuous  concave  densities  satisfies  the 
assumptions  of  Theorem  1 and  thus  its  rate  distortion  function  can  be  obtained 
explicitly. 

THEOREM  2.  Let  X be  a random  variable  with  density  p(x)  which  vanishes 

outside  the  interval  [a,b]  , -<»  < a < b < °°.  Suppose  p(x)  is  a continuous  concave 

function  on  [a,b]  and  there  is  an  at  most  finite  set  of  points  a = d^  < dj  < ... 

< d < d , = b(m>0)  such  that  on  each  |d-,d.  j=0,...,m,  p(x)  is  differen- 

m m+1  1 j j+1  J * 

tiable  and  its  derivative  is  absolutely  continuous.  Then  the  rate  distortion 
function  of  X is  given  by  (12)  and  (13)  . 

Proof : Since  p(x)  is  concave,  p"(x)  s'  0 and  p'(x)  S p'(x).  Also  / p(t)dt  > 0 

x0  a 

For  suppose  / p(t)dt  = 0 for  some  x^(  > a.  Then  p(t)  = 0 for  each 


J 


for  x > a. 


S i ii cc 


i 


tt  [ a , x | by  continuity  of  p.  Thus  p'(t)  0 for  each  t.  [n,x()|. 

p'(t)  p^(t),  we  have  p'  (t)  ■ U for  each  t.  [a,b|.  This  implies  p(t) 

t [a,b|  which  is  a contradiction.  Similarly  / p(t)dt  > 0 for  x < b. 

x 

only  assumption  left  to  be  verified  in  Theorem  1 is  (B) . 

If  p(b)  j-  0,  then  it  is  clear  from  (8)  that  k^(x)  -►  as  xth. 
now  that  p(b)  = 0.  Then  by  l'Hospital's  rule 


= 0 fo  r 
Thus  the 

Suppose 


lim  K (x)  = l im  p'(x)/p[x)  . 
xth  xtb 

We  will  show  that  p'(b)  < 0 and  thus  Kj(x)  ->  +oj  as  xtb.  By  the  concavity  of 
p(x),  p'(x)  is  a non- increasing  function.  Suppose  p'(b)  > 0.  Then  p'(x)  e 0 
for  all  x<  [a,b]  for  which  the  derivative  exists.  Thus  p(x)  is  non-decreasing 
Since  p(b)  = 0,  this  implies  that  p(x)  0 for  all  xr[a,b|,  which  contradicts 
fact  that  p is  a density.  Hence  p'(b)  < 0. 

The  proof  of  K?(x)  +°°  as  xfa  is  of  course  similar. 

It  should  be  noted  that  it  can  also  be  shown  that  Kj(x)tro  as  xth  and 
K-.lxlt™  as  xta  monoton  ical  ly.  [] 


the 


COROt.lARY  2.1.  Let  X be  a random  variable  with  continuous  probability  density- 
function  p consisting  of  line  segments  and  vanishing  outside  a finite  interval. 

Then  the  rate  distortion  function  of  X is  given  by  (12)  and  (13)  if  and  only  it 
p is  concave. 


Proof:  It  follows  from  Theorem  2 that  if  p is  concave  then  its  rate  distortion 

function  is  given  by  (12)  and  (13). 

Now  suppose  that  p is  not  concave.  Then  there  exist  two  adjacent  line  segments 
such  that  the  left  derivative  at  their  common  point  is  smaller  than  the  right 
derivative.  Hence  for  each  s,  C!  (y)  in  the  proof  of  Theorem  I is  not  a probability 
distribution  function  and  thus,  by  Theorem  A,  the  parametric  expressions  (12)  and 
113)  do  not  give  the  rate  distortion  function  of  X.  1 1 


T rape  .‘.o  id  density  (see  Append i 


Example  3. 

t)  < c < a and 


\ 1 for  the  derivation).  If 


P C x ) 


(a+c) ~ 

(a- | x ) )/(a“-c“) 


c 


c 


a 


( j 


then  for  0 < P $ (a-c)  (a+2c)/3(a+c) 

R(P)  = 2cos"[^  + 3COS  * (-30/^1“- c~)  ] - (a+3c) /2 (a+c) 

- dn(2/ (a-c)/ (a+c)  cos[-~  + -*cos  1 (-3l>/*^a2-c2)  ] ) (37) 

and  for  (a-c) (a+2c)/3(a+c)  < D < l>max  = (a3-c3) /3(a“-c“) 

R ( f) ) = -w(D)  - ? n ( 1 -a) ( D ) ) (38) 

->  i i 

where  w(D)  = l l-4D/(a+c)+(a-c)“/3(a+c)  1“’  . 

Theorem  1 is  also  valid  when  the  support  of  p(x)  is  not  finite.  The  result 
is  stated  in  the  following: 


THEOREM  1' . Let  X be  a random  variable  with  density  p(x).  If  p(x)  satisfies 
all  assumptions  in  Theorem  1 with  < a < b < +“',  then  the  rate  distortion  func- 
tion of  X is  given  by  (12)  and  (13)  with  < a < b < +<*>. 

Theorem  1’  is  an  improvement  of  Theorem  3 in  [Tan  and  Yao,  (1975)).  Here 
we  no  longer  require  the  monotonicity  of  lC(x),  i=l,2,  and  we  allow  p'(x)  to  have 
a finite  number  of  discontinuities  instead  of  a single  discontinuity  at  p. 

The  following  (known)  result  can  he  used  along  with  Theorem  1 and  l'  in 
evaluating  the  rate  distortion  functions  of  certain  random  variables. 

THEOREM  B.  If  the  random  variable  X.,  has  distribution  function  F.,  and 
distortion  rate  function  D. (R) , i=l,2,  then  for  all  R > 0, 

cx> 

|t)1(R)-n2(R)|  s / | Fj  ( t ) - F\  ( t ) | dt  . (39) 


th  -a 


Thus  if  F F -*•  0 in  I..  , or  if  1 > I weakly  and  i:  , I have  finite  mean.-,, 

n I 11  n 

P (R)  *■  l>(R)  uniformly. 

1' roof : Corollary  1 of  (Cray,  Neuhoff  and  Schields  (1975)]  applied  to  i.L.d.  source 

with  distributions  Fj  and  F,  gives 

|n1(R)-D2(R) ! < p(Fj,F2)  . 

CO 

But  by  | Va  l lender  (1975)]  p"(  F j . 1 „ ) = / | F'j  ( t)  - 1C,  ( t ) | d t and  thus  (39)  follows. 

_(Xi 

Now  if  }•'  converges  to  F weakly  and  all  distributions  involved  have  finite  means, 
then  by  Theorem  2 of  | Dob  rush  in  (1970)],  we  have  p ( !-'  , F)  k 0 and  hence  P (R)  * D(R) 
uniformly.  U 

The  following  well-known  property  is  useful  in  connection  with  Theorem  B: 

If  a sequence  of  probability  density  functions  p converges  to  a probability  den- 
sity function  p almost  everywhere,  then  the  corresponding  sequence  of  distributions 
F converges  to  the  distribution  F of  p weakly.  Thus  by  letting  c *■  <*>  in  example 
1,  we  find  the  rate  distortion  function  for  the  double  exponential  density  on  the 
entire  real  line  (since  all  distributions  involved  have  finite  means),  i.e. 

R(P)  = -£n  oil),  0 < P < ct  = D . Of  course,  this  has  been  calculated  by  using 

max 

the  Shannon  lower  bound  method  fBerger  (1971),  p.  95 | . Note  that  the  double 
exponential  density  does  not  satisfy  assumption  (B)  of  Theorem  1',  while  the  trun- 
cated double  exponential  densities  satisfy  all  the  assumptions  of  Theorem  1. 

This  demonstrates  how  Theorem  1 and  B can  be  used  in  evaluating  the  rate  distortion 
function  of  certain  random  variables  and  following  are  some  further  examples. 

example  4.  (a)  Triangular  density:  betting  c40  in  (36)  we  see  that  the 

trapezoid  density  converges  to  the  triangular  density 

(a- | x | ) /a“ , | x | • a . 

Its  rate  distortion  function  is  (lien  found  by  letting  c*(>  in  (37):  for 


0 < L)  < a/ 3 - 0 


max 


R(l>) 


_ 2.4  rr  1 -1  3D, 

?COS  — + —COS  (-  — ) 

a 


.■>  .•> 


1 o ,,  . 4tt  1 -1,  31) 

- - ?n(2cos[  + -cos  (-  -•] 


(40) 


(b)  Uniform  density:  Letting  eta  in  (36)  we  see  that  the  trapezoid  density 

converges  to  the  uniform  density 


1/ (2a)  , |x!  < a , 


whose  rate  distortion  function  is  thus  found  by  letting  cla  in  (5S):  for 


0 < 1)  < a/2  = D 

max 


R 


(Dj  = L-  ?P-  - in(l-^l-  2£  ) 


(41 


Note  that  the  triangular  and  uniform  densities  satisfy  all  assumptions  of  Theorem  1 
and  thus  (40)  and  (41)  could  be  obtained  directly  from  (12)  and  (13).  (41)  was 
first  given  by  Tan  and  Yao  in  [Cray  (editor)  (1974)]. 


III.  BOUNDS  TO  RATE  DISTORTION  FUNCTIONS 

In  this  section,  bounds  are  derived  for  rate  distortion  functions  of  random 
variables  whose  densities  do  not  satisfy  all  the  assumptions  in  the  theorems  of 
Section  II.  Examples  are  then  given  comparing  these  bounds  with  the  Shannon  lower 
bound . 

THEOREM  3.  Let  X be  a random  variable  with  probability  density  function  p(x) 
satisfying  the  assumptions  in  Theorem  1.  Let  Xj  be  another  random  variable  whose 
probability  density  function  p^ (x)  vanishes  outside  the  interval  [a,b],  and  p^(x) 
has  at  most  a finite  number  of  simple  discontinuities . Then  a lower  bound  for  the 
rate  distortion  function  of  X^  is  given  parametrically  in  s by 

I s|  Vs 

R[  (0S)  = -H  (pV  J.n  V - J.n(p(p-a  ) ) / Pj  (x)dx 

U+b  b 

- / ' Pj  (x)Hn(ep(x))dx  - «,n(p(u+bs))  / p (x)dx  (. 

M-as  pft’s 


w 


(43; 


M - a 


p+l> 


/ S(y-as-x)pj  (x)dx  + ,-T  / s Pj  (x)dx  + / (x-p-b.Jpj  (x)dx 
a ' I * ;j  - a p+b 


to 


b Pj(x) 

where  H^fp^j  = / Pj(x)£n  p'(-~xy  *s  the  generalized  entropy  of  p ^ with  resp < 
p [Pinsker  (1964) , p.  18];  p is  the  median  of  p and  a.  and  are  related  to  s by 
(10)  and  (11). 


Proof : Since  p(x)  satisfies  the  assumptions  of  Theorem  1,  A^  (x)  given  by  (30) 

satis  fies 

b 

Cs(>')  = / As (x)p(x)exp(s | x-y  | )dx  < 1 , for  all  ye[a,b]. 
a 


Now  define  a[^(x)  by 


Then  A^l^(x)  also  satisfies 
s 


A^Wp^x)  = As(x)p(x)  . 


(44) 


Cs(y)  = / A^ 1 ^ (x)pj (x)cxp(s | x-y | )dx  < 1,  for  all  yc[a,b]. 


According  to  Theorem  A,  A^^(x)  yields  a lower  bound  to  the  rate  distortion  func- 
tion of  Xj , i.e. 


sup(s0  + / p^(x)S,n  A^^(x)dx) 


s^O 


Let  R.  (D,s)  = sO  + / p (x)jin  A^(x)dx  and  let  d.,  j=l,...,n,  a = d.  < d 
L IS  J U 


< d(i  < = b be  the  points  where  Pj(dj)  has  simple  discontinuities.  Then 

from  (44),  (30)  and  (A. 5),  p^(x)£n  A^^(x)  is  continuous  both  in  s and  x and  its 
partial  derivative  with  respect  to  s exists  for  each  xe[a,b]  and  s < 0 and  is 
bounded  by  a constant.  Tlius 


9RL(0,s) 


b |i,  ( x)  3A^ 1 * (x ) 


3s 


d , n 1 d . . 

» i>  *f  / *♦  i !■"  ■ n m 

a j=l  d.  d A1  '(x) 
J ns 


H S 


u\  . 


3Rj (D.s) 

Setting Tj— = 0 and  substituting  (44)  in  the  ihove  expression,  we  have 


1 


K = - / 


b p j ( x J 3A^(x) 

TTxT  “37 


— dx  . 


Thus  for  each  fixed  D,  if 


32Rj  (P,s)/3st'  < 0 for  all 


< 0 


(45) 


then  Rj  (l>,s)  as  a function  of  s is  concave  and  its  supremum  is  achieved  by  the 
point  s satisfying  (45),  i.e. 

RL<-D,sD)  = sup  rL(D's)  ’ 
ssO 

Whether  or  not  (46)  is  satisfied,  R^(D  ) = R^(D  ,s)  along  with  (45)  provide  the 
parametric  expressions  (in  s)  of  a lower  bound  of  the  rate  distortion  function  of 
Xr  Substituting  (A. 5)  into  (45),  we  obtain  (43).  Substituting  (45)  and  (30) 
into  (4),  we  obtain  (42).  Note  that  the  generalized  entropy  of  pj  with  respect  to 
p has  the  property  H^(p^)  > 0 with  equality  if  and  only  if  p(x)  = p^(x)  a.s. 
[Pinsker  (1964),  p.  19 1 . a^.  and  b are  related  to  s by  (10)  and  (11).  It  should 

be  clear  from  (42)  that  R{  is  useful  only  when  H (p^)  < OT.  □ 


Clearly  Theorem  3 is  also  valid  when  the  support  of  the  densities  is  not  finite. 


THEOREM  4.  for  each  fixed  s < 0,  (D  ) gi^ren  in  Theorem  3 is  equal  to 
R(O^)  if  and  only  if  there  exists  a probability  distribution  function  Q whose 
total  probability  is  concentrated  on  (a  subset  of)  [a,bj  and  is  such  that 

b 


P,  (*) 


Is  e 


-sfp-a^-x) 


£l*i  / eslx-yldQs 


2p(p-as) 

¥/VMq.1c,-)  , 

a 

s(p+b  -x)  , 

I s [ e p(x)  t ,s|x- 

'"2p(vi>b  ) 1 


(yi 


‘clQ  (v) 
s 


xe[a,U-a  ) , 


xe [u-as ,p+bs | , 


xe  (p  *-b  ,b]  , 


(47) 


when.'  ns  and  b are  given  by  (10)  and  (11),  and 


/ As(x)p(x)e\p(s|x-y| )dx  = 1 a.e.  [dQj  . 


Proof:  Suppose  the  assumptions  are  satisfied  for  a given  s50.  Substituting  (30) 

into  (47)  and  using  (44) , we  have 

b m b 

Pl(x)  = As(x)p(x)/  exp(s|x-yj)dQs(y)  = J (x)p1(x)f  exp(s|x-y | )dQg (y) . 
a a 

If  p^(x)  i 0,  then  we  have 

rn  _ -I  ^ 

[\(  J(x)]-  = / exp(s|x-y|)dQs(y)  . (45 

a 

If  pj(Xg)  = 0 for  some  Xq€ [a,b] , then  from  (47),  Pj(Xq)  = p(x^)  = 0 for  some 
Xq£  [a.y-a^. ) u(y+bg  ,b] . In  this  case,  we  can  define  A^(x)  by  (49).  Therefore 
Rj  (Dg)  = R(Dg)  by  Theorem  A.  Conversely,  for  a given  s < 0,  suppose  R^(Ds)  = R(D  ). 
Then  A ^ (x)  achieves  the  supremum  in  (2)  and  hence  it  satisfies  (3),  i.e. 

(49),  and  is  such  that  C (y)  = 1 a.e.  [dQg(yj]  for  some  probability  distribution 
Q..  Substituting  (44)  and  (30)  into  (49),  we  obtain  (47).  □ 

It  would  be  of  interest  to  compare  the  lower  bound  of  Theorem  3 with  the 
Shannon  lower  bound  which  is  now  computed  for  densities  which  vanish  outside  the 
interval  [a,b] , -°°  < a < b < °°. 

THEOREM  C.  Let  X be  a random  variable  with  probability  density  function  p(x) 
which  vanishes  outside  [a,b]  , -00  < a < b < °°.  Then  the  Shannon  lower  bound  to 
R(D)  of  X is  given  parametrically  in  s<0  by 


RsL(Ds)  = h(p)  - | s | (b-a)/2k(s)  + £n|s[/2ek(s) 

n„  = Js  j"1  + (b-  a)  /2k  (s) 
b 

where  h(p)  - - [ p(x)7ri  p(x)dx  and  K(s)  - I - exp(|s  | (b-a)/2)  . Moreover, 


R (l>)  R(l>)  tor  all  I)  with  R(D)  > 0 uni 


p(x)  - |s | exp(s | x- (a+b)/2| )/2k  (s)  , x [a,b|  , 


(52) 


in  which  case* 


RS1 (n0)  = R(Dp)  at  the  point  with  slope  s. 


I’ roof:  L.et 

( k.  if  p(x)  > 0 
As(x)p(x)  = j S 

( 0 otherwise  . (53) 

from  condition  (1),  since 

b 

Cs(y)  = K J exp(s | x-y | )dx  , ye[a,b]  , 

' a 

one  can  take 

Ks  = s/2 [exp (s/2 (b- a) ) - 1]  . 


Theorem  A and  a simple  calculation  yield  (50)  and  (51). 

Now  from  Theorem  A,  R (D)  = R(D)  at  a point  1)  with  slope  s,  if  and  only 
if  C (v ) = 1 a.e.  [dG  ].  For  the  Shannon  lower  bound,  it  can  be  shewn  easil\ 

that  for  each  s 0,  C^(y)  = 1 if  and  only  if  y = lj(a+b) . Hence  R (D)  = R(I')  at 

0 with  slope  s if  and  only  if  puts  its  total  probability  mass  at  h(a+b)  in 
which  case 

_1  b 

[\SU)J  = / cxp(s|x-y|)dGs(y)  = exp(s  | x-Jj(a+b) ) ) . (54) 

a 

(52)  now  follows  from  (53)  and  (54)  and  can  be  verified  as  a density.  Hence  there 

is  one  s » 0 such  that  at  the  point  l>0  with  slope  s,  we  have  R^  (0^)  = R(Hq). 

For  all  other  densities  p which  vanish  outside  |a,b],  we  have  R (H)  < R(D)  for 
all  I)  ■ 0 such  that  R(H)  > 0.  [1 


Fxample  5.  I.et  p(x)  be  the  uniform  density  on  [a,b|.  Then  p(\)  satisfies 
all  assumptions  in  I'heorem  1.  l.et  p.(x)  be  a piecewise  continuous  densitv  defined 


on  |a,b|.  Then  by  applying  Theorem  3,  a lower  bound  tor  K(T>)  of  pj(x)  can  he 
found.  Evaluation  of  (42)  and  (13)  shows  that  (see  Appendix  5)  for  s < -(b-u) 

I . I - 1 

Is  I 

R(  (Ps)  = h ( p 1 1 + £n  | s | /2e  + / [p^a+tj  + p j ( 1>  - 1 ) ] d t (53  i 

, ,-i  ls;|_1 

= | s 1 - / t [p.  (a+t)  + p.(b-t)]dt  . (56) 

0 

In  (56),  if  for  a given  D,  |s|  is  not  single-valued,  and  if  a branch  of  |s|  can 
be  chosen  such  that  (46)  is  satisfied,  then  for  this  branch  of  |s|,  Rj  (L))  is  the 
best  possible  lower  bound  achieved  by  the  method  of  Theorem  3.  Note  that  condi- 
tion (46)  is  equivalent  to 

Pj (a+ 1 s I" 1 ) + Px  (b- | s | " 1 ) < |s]  (57) 

in  this  example  (see  Appendix  5). 

Ilie  lower  bound  of  Theorem  3 is  of  course  useful  when  Theorem  1 is  not 
applicable  to  pj(x).  As  an  illustration  we  now  calculate  the  lower  bound  of 
Theorem  3 when  Pj  is  the  truncated  double  exponential  density  of  Example  1.  In 
this  case  the  rate  distortion  function  of  p^  has  been  calculated  in  Section  II 
and  therefore  we  can  see  how  tight  is  the  lower  bound  determined  by  (55)  and  (56). 
Calculations  show  the  following: 

h (p)  1 - ? + Hn  2u(ct)  - ’jufct)  1 

z 

R|  ( r>s ) - - y + Hn|s|u(a)  + [2  exp(a/|s|)  - u-2|/2au(a)  |s|  > 2 (53) 

l's  = | s | 1 - [au(a)  | 1 [ ( | s | l-a' ^expfu/ | s| ) + a 1 | 
f>m.tx  ( j - (’■•  +«_1)exp(a/2)J/|l-exp(-a/2)  | 


where  u(a)  [exp(a/2)  - 1 |/a. 


Tor  the  Shannon  lower  bound  we  have 


s|  -0t)/2 


.>u  (t) 


+ in  2u(u)  + mi  v(s)  i v(s) 


P 


“si/V  ' 

ns  | s I 1 ( 1 v ( s ) ) . | s | > 0 (51)) 

where  v(s)  | s | /2 (exp( | s | /2)  - 1).  Curves  are  plotted  for  a - 0.1,  2 in 

figures  1 and  2.  In  general,  Rj  is  a better  lower  bound  than  K , except  in  a 

small  neighborhood  of  L)  where  R..  is  better.  As  a -»  0,  the  difference 

between  Rj  and  R^  becomes  larger  and  as  u *■  «>,  tlie  difference  becomes  smaller. 

It  can  also  be  seen  that  R^  is  a very  good  approximation  to  R.  If,  for  a > 0 

fixed,  we  plot  1)  as  a function  of  |s|  as  given  by  ( 5 S ) , we  obtain  a curve  as 

shown  in  Figure  3.  Note  that  at  [s|  = 2,  D =0  . Also  0 achieves  its 

1 1 s max  s 

maximum  at  some  point  |s(J  > 2,  and  P is  a decreasing  function  for  all  |s|  > lsL)l- 
It  follows  (as  it  is  easily  checked  analytically')  that  for  all  |s|  > !s  I, 

iii  max i 

condition  (57)  is  satisfied  and  thus  the  branch  of  Is!:  |s|  ">  |s  | gives  the 

ii  ii  i max i & 

tightest  possible  lower  bound  R (0^) . 

Another  lower  bound  for  the  rate  distortion  function  of  the  truncated  double 
exponential  density  can  be  obtained  by  using  the  truncated  Gaussian  density  instead 
of  the  uniform  density,  i.e., 

P ( x ) = K exp(-x“/2a")  , I x|  < F> 

where  K 1 = /2tF  [2$(l/20)  - l]  and  $ is  the  standard  normal  distribution.  Since 
p(x)  satisfies  all  assumptions  in  Theorem  1,  the  following  lower  bound  for  the  rate 
distortion  function  of  p^(x)  is  obtained  by  Theorem  3: 

R,  (»s)  = £n[|s|K(l-exp(-ct/2))/a] 

+ [{(So2)'1  - (l -a’ “a"2)  (1  t-a/2)  + £n  K - a2/2a2}exp(-a/2) 

- 1 2 

- 5,n  K + (l-(as+a  )/aa  )exp(-aai.)  ]/|  l-exp(-o/2)  ] 
l’s  = [ | s | 1 + (as-b-a  1)exp(-a/2)  - (|s|  *-ot  1 ) exp(  -aa^)  ] / | 1 -exp( -cx/2)  [ . 


The  relationship  between  a.  and  s < 0 is  given  by 


| s ! cxp[-a*/(2o2)]/{(<&((2u)-1)  - *(as/a) |/2r  a}. 

Numerical  calculations  show  that  this  lower  bound  is  slightly  better  than  the  one 
obtained  via  the  uniform  distribution. 

If  the  method  used  in  Theorem  1 is  applied  to  a discontinuous  probability 
density  function,  a A_(x)  5 0 may  be  found  satisfying  condition  (1)  whereas  a 
distribution  function  G__(y)  satisfying  (b)  and  (7)  may  not  exist.  In  this  case, 
using  the  above  mentioned  A , a lower  bound  for  the  rate  distortion  function  of 
the  discontinuous  density  can  be  obtained  by  (2).  The  following  example  illus- 
trates this  point. 


Example  6.  (For  the  derivation,  see  Appendix  6.)  Let 


p(x)  = 


1/4  -1  < x < 0 

3/8  0 < x < 2 


Then  for  0 < 1)  S 9/16,  we  have 


Rj  (0)  = -V4-5D  - £n(2-  /4-5D)  - % £n(54/625) 
and  for  13/24  < 1)  < 17/24  = , R(D)  itself  can  be  found  and  is  given  by 

R(D)  = - /l-(24D-l)/16  - Hn(l-  A- (2411- 1)  / 161  . 


Theorem  B may  yet  be  used  in  another  way  to  find  bounds  for  distortion  rate 
functions  of  discontinuous  densities. 


Example  7.  Let  p(x)  be  the  density  of  Example  6 and,  for  0 < e < 1,  consider 
the  continuous  approximating  density 


( 1/4 

x~/Uv  “ + x/Se  + 5/16 


'/!(>-“  + x/8t.  + 5/16 


i 3/S 


- 1 x - f. 


-r  • x ■ 0 


0 • X • L 


t x < 2 


Note  that  p (x)  converges  to  p(x)  almost  everywhere  as  c ■*  0.  In  this  case  the 
R ([))  of  p;  can  be  evaluated  by  Theorem  1 and  for  r.  - 0.59567,  we  have 

0 (R)  -0.00567  < D.  (10  < D(R)  !>  (R)  +0.00567 
£ L*  t 

where  Dj  (R)  Is  the  inverse  function  of  Rj  (0)  in  lixample  6.  (lor  the  evaluation 
of  D (R) , see  (Leung  (1976)  |.) 


IV.  APPKNDIX 


Appendix  1.  0 erl vat i on  of  (26 ) 

Substituting  (25)  into  (20)  we  have 


S.+n-l  | | -) 

I [p(t)-s~ “p"  (t)+C  5(t-p+a  )+C,  <S(t-p-b  ) + l D.  <“)  ( t-d  • ) |es  1 x‘  lit  r~a 

V ’ ' ~ j =1  J J 1 ' 1 


p(x)  , 


Suppose  Xt(d,  ,d.  1,  where  p-a  < d.  < p+b  . Then 

K k+ 1 s k s 

p+b  _9  , , s (x-p+a  ) S(p+b  -x) 

/ s(p(t)-s  p"(t))c  1 ‘dt  + C c + C e 

Vi-  a 

k s(x-d.)  £+n-l  s(d.-x) 


l n, 


p(x)  . 


a+b  t i „ i do  ?-+n-2  d P+b 

/ s(p(t)-cV(t)^s|x'c|Jt  = /*.  i p*1  * r 

J-as  p-as  j=»  d.  de+nl 

(pit ) s -p"(t))c 

...  -2..„,t^..s(x-t)  


s x t 


Since  for  each  st  (-on,-2pfp) ) , (p(t)-s  “"p"  (t))e  is  a.e.  |l.eb.j  on 


1 s(\-t)  . . 

|p-a.,d,|  equal  to  the  derivative  of  -s  (p'(t)  + sp(t))e  wane 

p:  ion  ( A)  is  absolutely  continuous,  we  have 


(A.l) 


(A.  2) 


h bv  nssin:- 


1 


/ (p(t)-s  V(t))es(x  f\lt  - - L |e  ^p'Ct)  + sc  *■  p ( t ) J 


c , “St 


sx  -sd  -sd.  -s(u-a  ) 

— -[e  p^d^)  + so  'pfd^)  - e S p|(p-as)  - so 

s“ 


s(p-a  _) 


Similarly,  for  2,  < j < k- 1 , 


f (p(t)-s"2p”(t))e'S(x_tldt 
d, 

sx  -sd.  -sd.  sd.  -sd. 

= - ^~2  Ie  J+  P^Cdj+1^>  + se  J+  p j + - e Jp+(dj)  - se  Jp (dj ) ] ; 


/ (p(t)  - E^V^dt 


-|p'(x)  + PO)  + s “[p^(dk)  - | s | p(dj_)  ]e 


s(x-dR) 


for  k + 1 < j < 2+n-2, 


/ (p(t)  - s'2p”Ct))eS(t'X)c 


s 2p'  (x)  + |-|-j-  pCx)  - s'2[|s|p(dk+1)  +p;(dRil)|e 


s(dk+rx) 


/J  + 1(p(t)  - s”2p"  (t) ) eS ('t_X^dt 


-sx  sd.  sd.  sd.  sd . 

r p ^ r,  I Tfl  ^ _ en  i rt  f A 1 - o ^ + cp  i T 


~2  [e  J+  pUdj  + 1)  - se  J+  p(d^+1)  - e Jp|(dj)  + se  Jp(dj)|  ; 


/ 5 (pCt)  - s-2p"(t))es(t'x)dt 

d„ 


2+n-l 


-sx  s (y+b  ) sCy+bs)  sd 

— 2“fe  ' p;Cu+bs)  - se  p(M+bs)  - e P^Jt+n-P 

s ' . 

Stl2+n- 1 . . 

+ se  Pfd^.j)]  • 


Substituting  the  above  expressions  in  (A. 2)  and  combining  similar  terms,  we  have 


/ s(p(t)  - s ‘"p" (t) )eS  x dt 


|"-j-  p(x) |[  |s|p(|i-as)  - P|(P-asHe 


s(x-y+as) 


-,[  | s | p(p<-b  ) * (. u f b ) |o 


f s ( x - d . ) S (,! . 'j 

- } [ [>•  (d  ) - p ' (d . ) |e  J }'  s -|p'(d.)  - p'(d  )|e  J 

.i  2 J J j=k+l  ' J + J 

liquating  coefficients  in  (A.l)  we  obtain  (26). 

Appendix  2.  Calculation  of  >s(xj  (i.e.  (30)). 

From  (6),  we  have,  for  xe[a,P-a  ] 

_ . p + b 

[A  (x)]  = / Sexp(s(t-x))dG  (t) 

^ s 

M*as 

and  substituting  (25)  and  (26),  we  obtain 

d„  n,  t d.  , p+b 

l 2+n-2  j+1  s 

[A(x)]‘  = { / + I / + / }[p(t)-s  “p"(t) jexp(s(t-x))dt 

p-a  j=2  d-  d„  , 

M s J 2,+n-l 

+ exp(y-as-x)[|s|p(p-as)  - pj(p-as)]/s2 
2+n-l 

+ I s tp' (d  •) -p' (d • ) ] exp (s (d . -x) ) 

j =£  J J J 

+ s [)s|p(p+bs)  + p^ (p+bs) ]exp(s (p+bs-x) ) . 

fhe  integrals  in  the  above  expression  have  been  calculated  in  Appendix  1 (except 
for  minor  adjustments  in  the  exponents).  Thus,  using  results  similar  to  those  in 
Appendix  1,  we  have 

As(x)  1 = 2exp[s(u-as-x)]p(p-as)/|s| 

and  the  expression  for  As(x),  xefa.p-aj,  given  in  (50).  The  calculation  of 

A (x)  for  xe[p+b  ,b]  is  similar.  For  xe [p-a  , p+b  ] we  have 
^ s s 

As(x)  = cts(x)/p(x)  = | s j / 2p(x) 


from  the  solution  of  (17). 


■ 


A ppendix  3.  1 )er i vat_i on_  of  (12 ) and  ( 13). 

The  relation 

b 

Ds  = - J (P(x)/As (xj ) f3Xs (x)/3s ld\ 
a 

is  proved  as  in  [Tan  and  Yao  (1975)  J and  thus  its  proof  is  omitted  here.  Il:e 
calculation  of  3A  (x)/3s  is  given  in  the  following:  From  (50),  we  have  for  all 

xc (a,y-a  ) 


3\.(x) 

3s 


3p(p-as) 

— ^ — — ]exp(-s(p-as-x)) 


2Pto-*s>  2p2(p-as) 

3a 

2p(p-as'j'ts  “37  ’ (y-as-x)]exp(-s(u-as-x))  . 


(A.  3) 


Differentiating  (21)  with  respect  to  s,  we  have 

das  2 9Pk*-as) 

-pOJ-as)-57  = -s  [s  g^-—  - p(p-as)  I 

and  thus 


da 


3p(p-as) 


ds  spfy-a^)  3s 


(A.  4) 


Substituting  (A. 4)  into  (A. 3),  we  obtain 


3Xs(x) 

3s 


(U-a  -x)A  (x)  for  all  xe(a,p-a  ) 

S -5  5 


The  calculation  for  xc(y+bs,b)  is  similar.  Thus 


3As(x) 

3s 


-(U-as-x)As(x) 


FT 


KM 


[ - (x-p-bs)As(x) 


x£ (a,y-as) 
xc  (p-as,p+b  ) 
xc(y+bs,b)  . 


(A. 5) 


Substituting  (A. 5)  in  the  expression  for  D we  obtain  (15).  From  (4)  and 


(A. 5)  we  have 


i i M-a  i'-a 

R(p  ) si)  +•  f ii  |S;'  + / Ss(x-U  + :t  )p(x)dx  - £u(p(y-a  )!  / 'p(xldx 
's  a ' a 

y *b  b b 

/ Sp(x)£np(x)dx  + / s (y+|>  ,-x )p (x)dx  - £n(p(y+b  ))  / pfxjdx  . 

u- a y+b  ' ’ y»b 

s s s 

Substituting  (A. 6)  into  the  above  expression,  we  obtain  (12).  finally,  since 
l) 

I)  = inf  [ |x-y|p(x)dx,  it  is  easy  to  verify  that  the  infimum  is  achieved 
max  1 1 . 

y a b 

when  y = y,  the  median  of  p(x)  , and  thus  1)  = / |x-y|p(x)dx. 


Appendix  4 . Per  i vat  ion  of  E xamp  le_  5 . 

Here  we  only  sketch  the  calculations  in  Example  3 and  omit  the  lengthy  details. 
Since  p(x)  is  symmetric  about  x = 0,  we  have  a^  = b^  and  (11)  gives  the  relation- 
ship between  |s|  and  a^: 


I s 1 = 2/(a-as) 


for  a / [c,a] 


s|  = 2/  (a+c-da^)  for  ase[0,c]  . 


IVe  now  distinguish  two  cases  (i)  and  (ii). 

(i)  If  | si  > 2/(a-c)  , using  (A. 7),  (12)  and  (13),  we  obtain 

R(Ps)  = £n*-s^--cl  a 2/s2(a2-c2)  - (a+5c)/2 (a+c) 

D = - 4/3 1 s | 3 (a2-c2)  . (A 

5 !sl 

The  parameter  s in  (A. 9)  can  be  eliminated  as  follows:  Substituting  x = 2/|s|(a+c) 

into  (A. 9),  we  obtain  the  following  cubic  equation: 

x3  . 1 x + 9i*z£l D = o. 

a+c  ( a + c ) ~ 

The  solutions  of  this  cubic  equation  consist  of  three  unequal  real  roots 
x^  = 2/(a-c)/ (a+c)  cos  cos  ^ (-3P/v/a“-c“  + 2kir/3)  | , k=0,l,2  . 


It  can  be  shown  that  only  x satisfies  the  condition 


| s | > 2/ (a-c)  . 


The  ro  to  re 


l\(  D)  -fn  x -,  + x~(a+c)/2(a-c)  - (a+Sc) /2 (a+c) 

which  gives  the  expression  in  (57)  for  I)  < I)  •'  — a JL 

.•>  ( a+c) 


(ii)  It'  | s | 2/(a-c),  using  (A. 8),  112)  and  (15),  we  obt; 


a in 


R(IM  - Hn|s | (a+c)/2  + 2/|s|(a+c)  - 1 

1>S  = Is!’1  - l/s’fafc)  * (a-c) “/ 12 (a+c ) . 

Eliminating  s in  (A. 10),  we  obtain  the  expression  in  (58)  fur  (a-c) (a+2c)/5(a+c) 


( 


< 0 < (a''-c‘>)/5(a~-c“)  0 


max 


Appendix  5.  Derivation  of  hxample  5. 

In  Theorem  5,  let  p(x)  (b-a)  * for  x • [a,b|.  Then  from  (10)  and  (11),  we  ) 
as  - b^  = !2(b-a)  - |s|  1 , p = ’Ab-a)  . 


A 1 so 


-H  Cp i ) = - / P1(x)Hn(p1(x)  (b-a))dx  h(pj)  - £n(b-a) 
a 


i-l 


Thus  (42)  becomes 

Is  I aVS| 

Rj  (n^.)  = b(pj)  - ?.n(b-a)  + + fn(b-a)  / pj(x)dx 

a 

1 


b-  | s I b 

- / p (x)(Ln  r— 7-  dx  + tn(b-a)  / p (x)dx 

• i o-a  i i i ~ * * 

a+ I s | “ 1 

-1 


b-  s 


h(pt)  + in  + / 


b-  s 


a+  i s | 
i-l 


i -1^1 


p, (x)dx 


i i a+ 1 s | n 

h(Pj)  + £n  2 -1  +/  Pj(x)dx  + / | p | ( x ) dx 

a b - | s | 


i-l 


= h (pj)  + ?.n  'yL  + / (Pj  (a  + t ) + pj  (b-t ) |dt  . 


A.  101 


a ve 


From  (43)  we  have 


a+M  1 , Ms|  b ! 

3S=/  (a+  rrr  -x)pi(x)dx  + [vr  J,  ipi(x)dx  + J,  l-i(x-b+T7r)pi(x)dx 

a 1 1 a + j s | t > - 1 s | 11 


J t (p  (at-t)  + p.  (h-t)  ) cl t . 


This  proves  (55)  and  (56). 

Now  condition  (46)  is  equivalent  to 


h p (x)  3“X  (x) 

^ X (x)  * 2 

a s 3s 


1 3Xs(x)  2 
Y.(x)  3s  ^ 'dx  " 0 


which  is  equivslent  to  (57)  when  A (x)  is  i-eplaced  by  (30)  for  p(x)  = (h-a) 


Apjien  d lx  6 . Deri  vat  i on  o f Example  6 . 

2 2 

Calculation  shows  that  p = 2/3.  Let  V's  = - a^ , — + h^]  he  the  support 

the  distribution  Cs (y) . 

2 

(i)  Suppose  - - a^  > 0.  This  case  is  similar  to  the  case  of  the  uniform 
distribution.  One  can  apply  Theorem  1 directly  to  get : 

R(D ) = tt4t  - 1 + «n  1M, 


, _ 1 3 J_ 

S TsT  8 1 s 1 2 24 


3 , , 3 

— < s < — 

4 1 1 2 


F.liminating  Isl,  one  obtaines  the  desired  expression  (61)  for  < D < = D . 

2 

(ii)  Suppose  -1  S -j  - a^  < 0.  Then  gs(y)  has  the  form  for  all  yeV__ 
gsf>')  = P(y)  + C 6(y  - | + a ) + C0  S'(y-O)  + C 6(y  - \ - h ) 

’ 1 ) 3 o <-  j 5 0)0  J o 


where 


, C- 
2,s 


5 1 . 

as  = 3 - ITT  ’ bf 


4 1 


Also 


from  these  expressions,  A^(x)  can  he  calculated  and  has  the  following  form: 


As(x)  = 


1 i ^ 

>!(-|  - as)  - J s ( x 

1 < 

2 | 

| s |e 

7 

2 1 

s | 

*2  _ 

3 

4 | 

|s| 

0 < 

5 

7 

4. 

- 

|s| (3  * bg ) + | s 1 x 

-> 

^ ■ 

s | e 

+ ! 

x • „ - a 
j s 


6 s 


It  remains  to  show  that  C (y)  S l for  y4V  . We  will  show  this  when  y > — + b 

s s 3 s 

2 

The  case  y < — - a is  similar.  Now 
3 s 


2 (2/3) -a 

C (y)  = / X (x)p  (x)exp(s  | x-y  | ) dx  = / s ~^\ 

-Is  -Is 


S |cxp(|s|  (-J  - as-y) 


0 (2/3)+b 

+ / j I s lexp(- I s J (y-x) )dx  + / y|s|exp(-|s) (y-x))dx 

(2/3) -a . 0 

y 2 

+ / \ |s|exp(- I s I (~  + b +2x-y)dx  + / \\  s | exp (- 1 s I (|-  +b J *y) dx 

( 2/3  i t-b  “ J v Z * s 

s 

" ^ exP  1 1 s U 3 +bs“>'^l  + +ls|-  -f|s|y)exp[|s|  (y  - b^) ) . 

Differentiating  C (y ) with  respect  to  y,  we  have 

£*(>')  = | s |exp[ | s | (y  - - bs)]f(y) 

where  f(y)  = |s|-  ^ - -~|s|y  - iexp[2|s|(i  + bs  - y) ] . Now  f • (y)  = 

I -J 

2 1 s I Co xp [ 2 1 5 1 ("  + - y)  | - l},  and  f * (y^)  = 0 if  and  only  if  = -^  + b^. 

2 

Since  f"(y())  = - s < 0,  f(y^)  is  a maximum.  But  f C >' q ) = 0 and  thus  f(y)  < 0 

? 

and  (!*  (y)  < 0 which  implies  that  Cc  (y)  is  non- i ncreasing . Since  ~ + b^  - 

2-  [ s | \ we  have  C (y^)  = 1.  Thus 


C (y)  < l for  all  v > \'  = t!  * b 

s ' 0 .*>  s 


31 

Since  g^_  (y ) is  not  a probabi  1 i t>  density  function,  the  A ^ ( x J found  above  cin  be 
used  only  to  provide  a lower  bound  to  the  R(D)  of  p(x)  . Thus  by  Theorem  A 

11,(1),  si  = - | s 1 1)  + 5/  ( 16  | s | ) + i In  2 | s | + (3/4)  f,n  (4 1 s | /5)  . 

Differentiating  R, (D,s)  with  respect  to  |s|  and  setting  R'(D,s)  = 0,  we  obtain 

ns  = | S r 1 - 5/(16|s|2)  , |s|  > 1 . 

Also,  R'^(D,s)  = 5/  (8 1 s | 3)  - |s|  1 < 0.  Eliminating  |s|  we  obtain  (60). 


1 


BIBLIOGRAPHY 

Burger,  T.  (1971).  Rate  Distortion  Theo ry , Prentice  Hall,  Inc.,  Englewood 
Cliffs,  New  Jersey. 

Dob  rush  in,  R.L.  (1970).  Prescribing  a system  of  random  variables  by  condi- 
tional distributions.  Theory  of  Probability  and  Its  Applications,  15: 
pp.  45S-486. 

Gray,  R.M.  (Editor)  (1974).  Source  Coding/Ergodic  Theory  Research  Seminar 

Lecture  Notes.  Stanford  University  Information  System  Laboratory  Techni- 
cal Report  No.  6503-2:  pp.  93-101. 

Gray,  R.M. ; Neuhoff  and  Schields  (1975).  A generalization  of  Ornstein's  d 

distance  with  applications  to  information  theory.  Annals  of  Probability, 

3:  pp.  3L5-32S. 

Leung,  11. M.  (1976).  Bounds  and  the  evaluation  of  rate  distortion  functions, 
Institute  of  Statistics  Mimeo  Series  £1057,  University  of  North  Carolina 
at  Chapel  Hill. 

Pinsker,  M.S.  (1964).  Information  and  Information  Stability  of  Random  Variables 
and  Processes,  Holden-Day,  Inc.,  San  Francisco. 

Tan,  H.H.  and  Yao,  K.  (1975).  Evaluation  of  rate-distortion  functions  for  a 
class  of  independent  identically  distributed  sources  under  an  absolute- 
magnitude  criterion,  IEEE  Transactions  on  Information  Theory,  IT-21: 
pp.  59-64. 

Val lender,  S.S.  (1975).  Computing  the  Wasserstein  distance  between  probability 
distributions  on  the  line.  Theory  of  Probability  and  Its  Applications,  18: 
pp.  824-827. 


